Exceptional Unitary Representations of Semisimple Lie Groups

Let G be a noncompact simple Lie group with finite center, let K be a maximal compact subgroup, and suppose that rank G = rank K. If G/K is not Hermitian symmetric, then a theorem of Borel and de Siebenthal gives the existence of a system of positive roots relative to a compact Cartan subalgebra so that there is just one noncompact simple root and it occurs exactly twice in the largest root. Let q = l⊕u be the θ stable parabolic obtained by building l from the roots generated by the compact simple roots and by building u from the other positive roots, and let L ⊆ K be the normalizer of q in G. Cohomological induction of an irreducible representation of L produces a discrete series representation of G under a dominance condition. This paper studies the results of this cohomological induction when the dominance condition fails. When the inducing representation is one-dimensional, a great deal is known about when the cohomologically induced representation is infinitesimally unitary. This paper addresses the question of finding Langlands parameters for the natural irreducible constituent of these representations, and also it finds some cases when the inducing representation is higher-dimensional and the cohomologically induced representation is infinitesimally unitary. Let G be a simple Lie group with finite center, let K be a maximal compact subgroup, and suppose that rank G = rank K. Let g0 = k0 ⊕ p0 be the corresponding Cartan decomposition of the Lie algebra, and let g = k⊕ p be the complexification. In this paper we investigate some representations of G first studied in [EPWW] that are closely related to a fundamental kind of discrete series representations of G. We are especially interested in the Langlands parameters associated to these representations. We begin with some background. In an effort to find unusual irreducible unitary representations of G in the case that G/K is Hermitian symmetric, Wallach [W1] studied “analytic continuations of holomorphic discrete series.” When G/K is Hermitian symmetric, p splits as the direct sum of two abelian subspaces p and p−, and k⊕p is a parabolic subalgebra of g. An irreducible representation τΛ of K leads via this parabolic subalgebra to a generalized Verma module that is a (g,K) module. If the highest weight Λ of τΛ satisfies suitable inequalities, this (g,K) module arises from a holomorphic discrete series representation [HC1]. Wallach [W1] studied the “scalar case,” in which τΛ is one-dimensional. By adjusting Λ on the center of k, he was able to move Λ in a one-parameter family. For values of Λ outside the range that yields holomorphic discrete series, he determined necessary and sufficient conditions for the unique irreducible quotient of the generalized Verma module to be infinitesimally unitary. Received by the editors June 19, 1996 and, in revised form, August 5, 1996. 1991 Mathematics Subject Classification. Primary 22E46, 22E47. Presented to the Society August 7, 1995 at the AMS Summer Meeting in Burlington, Vermont. c ©1997 American Mathematical Society